Understanding political knowledge through global expert surveys

Universitat Oberta de Catalunya (UOC)

Jordi Mas Elias

https://www.jordimas.cat/

Introduction

Introduction

  • In recent years, expansion of the expert survey methodology, a method for measuring complex political concepts (Marks et al. 2007).
  • On party ideology, recent datasets include over 1,000 observations and > 150 countries, e.g. V-Party (Lührmann et al. 2020) and GPS (Norris 2020).
  • But, to what extent broader coverage in expert surveys comes at the cost of reduced precision?

Objectives

  • Investigate whether experts exhibit greater disagreement when evaluating parties:
      1. in less democratic institutions;
      1. relatively small
  • Contribute to the literature on expert survey methodology.

Theory

Expert surveys (I)

Since social phenomena are complicated to measure, some strategies rely on surveying experts: individuals with specialized knowledge of one or more countries who can synthesize multiple sources of information (Hooghe et al. 2010).

Expert surveys (II)

Expert surveys face two primary types of error: systematic and random error (Maestas 2018):

  • Systematic (bias / accuracy): Non-independent assessments. Consistent in the experts’ responses, leading to a deviation from the true estimate. Very problematic because it skews the results.
  • Random (noise / precision): Independent, but different assessments. Variability in the experts’ responses that does not follow a specific pattern. Unproblematic if truly independent, but in most cases, the degree of independence is unknown.

Expert surveys (III)

Sources of random error (Budge 2000; Martínez-Coma and Van Ham 2015):

  • Individual cognitive and judgment biases (e.g. experts might rely in different heuristics, have ideological leanings or vary in their knowledge).

  • Political and social context under which the evaluation is made (e.g. autocracies can punish certain political orientations).

  • Some concepts, due its nature, are inherently complex (e.g. political ideology, government effectiveness, political polarization).

Theoretical arguments

Theoretical arguments (I)

Regarding context’s characteristics, random biases have been extensively explored in Europe (Bakker et al. 2014), but less outside of it. We can expect that the type of political regime affects:

  • Information asymmetries faced by experts: more information in democracies, less in autocracies.
  • Ideological leaning of the assessments: more leaning in autocracies, less in democracies.

Theoretical arguments (II)

About the unit’s characteristics, non-significant relationship between party size and experts’ disagreement (Marks et al. 2007; Steenbergen and Marks 2007). But:

  • More information (resources, media attention) on large parties than small parties.

  • But more ambiguity in catch-all parties - strategies more difficult to evaluate with precision.

  • Thus, more precision in “medium” parties?

Theoretical arguments (III)

But though small party units may lack of resources, some can be easier to evaluate if have enough stability and longevity in the party system. One can difference between:

  • Statewide small parties, often with shorter lifecycles, being more difficult to evaluate with precision.
  • Regional (small) parties, typically tied to specific regionally concentrated groups, more stable, being easier to evaluate.

Theoretical arguments (III)

About the policy dimension, not yet.

Methodology

Methodology (I)

  • Data from 168 countries and 10041 party-policy observations derived from the V-Party (Lührmann et al. 2020), CHES (Bakker et al. 2020) and GPS (Norris 2020).
  • Similar time period, aggregation, and dimensions (next).
  • Excluded assessments conducted with only one expert (from 2 to 84).
  • Controling for the fixed effects of the dataset.

Methodology (II)

Figure 1: Disagreement between experts

Methodology (III)

Figure 2: Percentage of votes to parties

Figure 3: Percentage of votes to parties

Methodology (IV)

  • Regional party (IV2): Defined as political parties that seek representation at the substate level (93 obs.). Data from Massetti and Schakel (2021) and Nordsieck (2019).

Figure 4: State and non-state wide parties

Methodology (V)

Figure 5: Continuous (V-Dem) and dichotomous (DCD) measures of democracy

Methodology (VI)

  • Policy dimensions: Three criteria: conceptual similarity; minimum pairwise correlation of 0.6; and data from at least two different datasets per dimension.

Methodology (VII)

Figure 6: Pairwise correlations grid

Results

Results (I): Bivariate analysis

  • Regime type and SD: Slightly negative, seemingly inverted U-curved relationship (specially for small parties).

Results (II): Bivariate analysis

  • Size and SD: Slight U-curved association, irrespective of the regime. Less disagreement for medium-sized parties.

Results (III): Bivariate analysis

  • Regional and SD: High disagreement on statewide parties, much less on regional parties, specially in autocracies.

Results (IV): Models

Results (V): Models

Results (VI): Models

Conclusion

  • Experts are more precise in democracies, though this can be potentially attributed to higher levels of economic development.
  • Democracies and autocracies seem to follow very different patterns, which need to be explored in depth.
  • If anything, party size follows a non-linear relationship with noise.
  • Experts show minimal disagreement when locating regional (small) parties, ceteris paribus, especially in comparison to small statewide parties.

Work to do

  • Explore non-linear relationships (party size and regime type).
  • Consider new controls: member of government, internal dissent, or policy-dimension FE.
  • Look at variations across policy dimensions.

References

Bakker, Ryan, Liesbet Hooghe, Seth Jolly, Gary Marks, Jonathan Polk, Jan Rovny, Marco Steenbergen, and Milada Anna Vachudova. 2020. 2019 Chapel Hill Expert Survey.” Chapel Hill, NC: University of North Carolina, Chapel Hill.
Bakker, Ryan, Seth Jolly, Jonathan Polk, and Keith Poole. 2014. The European Common Space: Extending the Use of Anchoring Vignettes.” Journal of Politics 76 (4): 1089–1101.
Boix, Carles, Michael K. Miller, and Sebastian Rosato. 2013. A Complete Data Set of Political Regimes: 1800-2007.” Comparative Political Studies 46 (12): 1523–54.
Budge, Ian. 2000. Expert judgments of party policy positions: Uses and limitations in political research.” European Journal of Political Research 37 (1): 103–13.
Castles, Francis G., and Peter Mair. 1984. Left–Right Political Scales: Some ‘Expert’ Judgments.” European Journal of Political Research 12 (1): 73–88.
Coppedge, Michael, John Gerring, Carl Henrik Knutsen, Staffan I. Lindberg, Jan Teorell, David Altman, Michael Bernhard, et al. 2021. V-Dem [Country–Year/Country–Date] Dataset v11.1.” Varieties of Democracy Project. https://doi.org/10.23696/vdemds21.
Graefe, Andreas, J. Scott Armstrong, Randall J. Jones, and Alfred G. Cuzán. 2014. Combining Forecasts: An Application to Elections.” International Journal of Forecasting 30: 43–54.
Hooghe, Liesbet, Ryan Bakker, Anna Brigevich, Catherine De Vries, Erica Edwards, Gary Marks, Jan Rovny, Marco Steenbergen, and Milada Vachudova. 2010. Reliability and Validity of the 2002 and 2006 Chapel Hill Expert Surveys on Party Positioning.” European Journal of Political Research 49: 687–703.
Lührmann, Anna, Nils Düpont, Masaaki Higashijima, Yaman Berker Kavasoglu, Kyle L. Marquardt, Michael Bernhard, Holger Döring, et al. 2020. Varieties of Party Identity and Organization (V-Party) Dataset V1.” Varieties of Democracy (V-Dem) Project.
Maestas, Cherie. 2018. Expert Surveys as a Measurement Tool: Challenges and New Frontiers.” In The Oxford Handbook of Polling and Survey Methods, edited by Lonna Rae Atkeson and Michael R. Alvarez. Oxford: Oxford University Press.
Marks, Gary, Liesbet Hooghe, Marco R Steenbergen, and Ryan Bakker. 2007. Crossvalidating data on party positioning on European integration.” Electoral Studies 26: 23–38.
Martínez-Coma, Ferran, and Carolien Van Ham. 2015. Can experts judge elections? Testing the validity of expert judgments for measuring election integrity.” European Journal of Political Research 54 (2): 305–25.
Massetti, Emanuele, and Arjan H. Schakel. 2021. From staunch supporters to critical observers: Explaining the turn towards Euroscepticism among regionalist parties.” European Union Politics 22 (3): 424–45.
Nordsieck, Wolfram. 2019. Parties and Elections in Europe. Books on Demand.
Norris, Pippa. 2020. Measuring populism worldwide.” Party Politics 26 (6): 1–21.
Steenbergen, Marco R, and Gary Marks. 2007. Evaluating expert judgments.” European Journal of Political Research 46 (3): 347–66.